Deep Learning of Invariant Spatio-Temporal Features from Video

We present a novel hierarchical and distributed model for learning invariant spatiotemporal features from video. Our approach builds on previous deep learning methods and uses the Convolutional Restricted Boltzmann machine (CRBM) as a building block. Our model, called the Space-Time Deep Belief Network (STDBN), aggregates over both space and time in an alternating way so that higher layers capture more distant events in space and time. The model is learned in an unsupervised manner. The experiments show that it has good invariance properties, that it is well-suited for recognition tasks, and that it has reasonable generative properties that enable it to denoise video and produce spatio-temporal predictions.